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ABSTRACT 

Summaries are presented of papers from a symposium 
entitled **The Evaluation of Complex Programs Using Participatory 
Evaluation'* that focused on a comprehensive school service program in 
a south Texas alternative high school. "Theoretical Framework and 
Objectives" explores the problems of evaluating such programs. 
Problems result from the complexity and flexibility of the programs, 
difficulties in evaluating collaboration, and the interdisciplinary 
nature of the endeavors. "New Research and Evaluation Designs for 
Comprehensive Service Models" by Jane Stallings provides an overview 
of the conceptualization of a research and evaluation agenda for 
complex models in general, and comprehensive service models in 
particular. "Increasing the Academic and Social Success of Children, 
Youth, and Families: The Options in Education Programs for High Risk 
Students" by Tabitha Gutierrez and Larry Kohler describes the Options 
in Education program created as an alternative for students at risk 
of school failure and dropout. The program offers comprehensive 
services to about 200 students from the primarily Hispanic American 
population of students in the district. "Implementing a Model for 
Participatory Evaluation: Findings from Three Studies" by Stephanie 
L. Knight and Karen Duhon describes the use of participatory research 
and evaluation to assess the Options in Education Program. 

"Evaluation of Complex Programs Using Participatory Evaluation" by 
Jane A. Stallings provides further information on the Options program 
and its evaluation. (Contains 16 references.) (SLD) 
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Evaluation of Complex Programs Using Participatory Evaluation 



Theoretical Framework and Objectives 

Many programs have sought to address the needs of children and youth who are 
currently underserved by focusing on collaboration among educators, parents, and 
community and service agencies (Dryfoos. 1994; Levin, 1994). Regrettably, information 
about these efforts is incomplete and fragmented. Furthermore, evaluation of 
comprehensive, collaborative services is often embedded in advocacy literature which 
describes the process for integration of services, but little information on program 
effectiveness beyond anecdotal support. For example, in a review of comprehensive 
programs for young children, Lopez and Weiss (1994) noted that not many programs 
conduct evaluations and few attempt to link programs to outcomes. While recent 
attempts to review existing literature have provided some degree of synthesis of studies 
addressing school-linked comprehensive .services (e.g., Chaudry et al.. 1991; Crow.son & 
Boyd, 1993; GAO, 1993), studies related to these efforts are "fugitive " at best (Knapp, 
1994, p.3). 

Methods to collect information about school-linked comprehensive services vary 
widely (Lopez & Weiss, 1994). Previous studies have often applied traditional 
experimental methods (e.g., Gomby & Larson, 1992); survey research (Kagan et al., 
1990); and multiple project comparative studies (Marzke et al., 1992; Wehlage, Smith, & 
Lipman, 1992). Case studies are the predominant type of assessment currently used, but 
they vary widely in quality and comprehensiveness (Crowson & Boyd, 1993). Typically, 
studies of school-based and school-linked services focus on single programs, often 
isolated projects not part of a comprehensive approach, which address specific issues 
such as teen pregnancy or health (Wang, Haertel, & Walberg, 1994). Most of these 
single intervention studies did not randomly assign comparison and intervention groups 
or match for potentially confounding characteristics; focused primarily on attitudes and 
knowledge and did not include behavioral, health status, or educational outcomes; and 
rarely included longitudinal designs (U.S. Department of Health and Human Services, 
1993). Furthermore, few studies have addressed cost effectiveness or benefits, an 
approach which currently is needed to determine the best use of scarce resources 
(Stallings, 1995). 

Several characteristics of school-linked comprehensive service programs make 
them very difficult to evaluate. In particular, the difficulty is intensified by (a) the 
complexity and flexibility of the programs making it difficult to adequately identify and 
characterize the independent variables of interest; (b) the difficulty in assessing quality 
and degree of collaboration; and (c) the interdisciplinary nature of the endeavor involving 
researchers from different fields who rarely communicate with each other (Kagan, 1991; 
Knapp, 1994). In order to address these challenges, alternative approaches to traditional 
evaluation have been suggested, including participatory evaluation involving stakeholders 
and clients (Weiss & Greene, 1992) and a developmental approach to evaluation which 
provides more formative feedback (Gray, 1993). 
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New Research and Evaluation Designs for Comprehensive Service Models 

Jane Stallings, 

Texas A&M University 



This paper provides an overview of the conceptualization of a research and 
evaluation agenda for complex models in general, and comprehensive service models 
specifically. As a result of two working conferences cosponsored by AERA and OERI. 
progress has been made toward formulating designs which capture the unique 
characteristics of complex, collaborative systems (U.S. Department of Education Office 
of Educational Research and Improvement, 1995). Previous literature identified issues 
focusing on engagement of diverse participants perspectives, characterizing the elusive 
independent variables, locating and measuring outcomes, attributing results to influences, 
and studying sensitive processes and outcomes (Knapp, 1994). Discussion of these issues 
at the working conferences resulted in a set of five recommendations including ( 1 ) initial 
in-depth conceptualization of the processes and outcomes of programs; (2) strategies for 
research and evaluation that are developmental, longitudinal, and participatory or client- 
driven; (3) use of context appropriate measurement tools; (4) u.se of evaluators with 
competencies in both qualitative and quantitative methodologies and experience in 
participatory methods; and (5) creation of means of dissemination which reach interested 
researchers, practitioners, and policy makers. These recommendations were used to 
create and implement an approach to participatory evaluation which seeks to (a) examine 
change in processes and outcomes over time; (b) provide programs with information 
which will enable them to build on strengths and address challenges; and (c) establish the 
relationships between processes and outcomes, including cost effectiveness outcomes 
which may influence decisions of program implementers and policymakers (Stallings, 
1995). 



Increasing the Academic and Social Success of Children, Youth, and Families: 
The Options in Education Program for High Risk Students 

Tabitha Gutierrez 
Larry Kohler 

McAllen Independent School District 

This paper describes the Options in Education program created as an alternative 
for students at risk of school failure and dropout. The program offers comprehensive 
services to meet the diverse needs of approximately 200 students from the primarily 
Hispanic population of students in the district. High risk conditions of the students 
served include family dysfunction, conditions associated with low socioeconomic status, 
early marriage responsibilities, teen pregnancy, substance abuse, and other barriers to 
employment or further education. The common bond is that all students have either 
dropped out of school or have accrued so few credits toward graduation that they would 
not graduate. In addition, they all chose to apply for entry into the Options program and 
committed to focus on attitude, attendance, and academics with the help of the program. 
Program features include individualized, self-paced instruction; summer and evening 
classes to provide flexibility in scheduling for students; concurrent employment 
opportunities; personal and group counseling; the services of an on-site social worker and 
school nurse; and job training in conjunction with the local community college. Pregnant 
teens receive their basic courses, plus classes in parenting, child development, job skill 
training, and/or college preparation. Child care is available while students are attending 
classes. 
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Implementing a Model for Participatory Evaluation: 

Findings from Three Studies 

Stephanie L. Knight 
Karen Duhon 
Texas A&M University 

This paper describes the use of participatory research and evaluation (Cousins & 
Earl, 1992) to assess the processes and outcomes of the Options in Education Program. 
University researchers met with school and district administrators; the district evaluator; 
school board members; parents; teachers; students; and community and agency 
representatives to determine the focus and nature of evaluation needed to understand the 
processes and outcomes of the program and to provide information to improve program 
efforts and results. Focus group and individual interviews were conducted to determine 
needs and priorities and to provide a description of the components of the program. As a 
result, three studies were designed, involving stakeholders in formulation of the questions 
and variables addressed as well as data collection and analysis. The first study focused 
on the follow-up of program participants since 1991. Surveys were mailed to 
approximately 300 former students, with a return rate of almost 40%. Analysis of the 
responses indicate that 85% of respondents either graduated or received a GED, 
compared with a 65% graduation rate for this population nationally. Furthermore, 84% 
were either employed or enrolled in higher education or vocational programs. Using the 
data gathered in the first study plus information on the costs of program components, cost 
avoidance analyses were conducted (e.g., Bruner, 1993). For example, information on 
rate of return for the teen pregnancy program based on program costs and the rate and 
costs associated with low birth weight babies revealed an annual rate of return of 20% for 
this component of the Options program. 

The second and third studies involve comparison of a sample of Options students 
(n=85) with a comparison group (n=77) that did not participate in the program. The 
comparison group consists of students who applied and qualified for inclusion in Options 
but because of program capacity limitations were not accepted. The two groups were 
compared using course credits accrued, attendance, courses passed, and Texas 
Assessment of Academic Skills (TAAS) results. The Options group performed 
significantly better on all measures than the comparison group. 

The third study was designed to equate possible differences between the Options 
and comparison group in the previous study. Based on the hypothesis that students 
accepted to the program might differ from those denied based on attitudes or motivation, 
a study was planned which would match experimental and comparison students on these 
attributes. Therefore, the Piers-Harris Self Concept Instrument was administered to the 
Options group. However, attempts to locate an adequate sample of denied applicants 
were not successful. Examination of school records revealed that about 50% of the 
comparison group had dropped out of school, were in jail, or reported moving to other 
districts. Approximately 20% had subsequently been admitted to Options or another 
alternative school in the area. Only six percent had graduated or received their GED. Of 
the original 77, only 15 students were available for comparison. 
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Evaluation of Complex Programs Using Participatory Evaluations 

Session 27.41 
by 

Jane A. Stallings 

Our symposia entitled The Evaluation of Complex Programs Using Participatory Evaluations 
will focus on a comprehensive school service program in a south Texas alternative high school 
and report preliminary outcomes. I will give an overview of the need for Comprehensive School 
Linked Services. Dr. Stephanie L. Knight, Texas A&M University, will present New Research 
and Evaluation Designs for Comprehensive Service Models.- Karen Duhon, Texas A&M 
University, and John Kohler, McAllen Independent School District, Texas, will present 
Increasing the Academic Success of Children, Youth, and Families: The Options in Education 
Program for High-Risk Students. Tabitha Gutierrez, of McAllen Independent School District, 
will be the discussant. 

The Comprehensive school linked service movement has been gaining momentum as it 
becomes clear to more and more of the urban and rural educators and service providers that such 
partnerships among providers holds the most promise for the success of their clients. It is an 
accepted fact that the number of low income families not receiving health care has significantly 
increased during the 1980’s and 1990's (500,000 families). 

The Plight of Our Children 

Cheri Hayes, executive director of the Finance Project wrote in the October 1993 Prospectus 
that, despite annual public expenditures approaching $350 billion, on education and services 
nearly a quarter of U.S. children are poor and live in families and communities that cannot meet 
their basic needs. Despite the broad consensus about the National Education Goals in the late 
‘80s and early ‘90s, and even with strong indications of public and political eagerness for a 
renewed emphasis on domestic social policy, fundamental reforms have been slow to materialize 
in the nation’s education, health, welfare, public housing, and social service systems. 




There can be little wonder as to why educational interventions based upon the body of 
teaching and learning research from the 1970s and 1980s, have had so little lasting effect for 
children at risk of school failure. The answer has taken a long time to emerge, but it is 
unequivocal: We were not considering the multitude of intervening variables. For example, 
although we knew that parents were crucial to child learning and that some children depended 
on the food they received at school for physical sustenance, we rarely included these variables as 
we assessed school success or school failure. It’s clear now that when we assessed interventions, 
we were not using all the playing pieces. We did not ask in a systematic way “What is required 
for children to come to school ready to learn, so that their chances for success can be optimized? 
What are the factors that prevent the best, most creative teachers from succeeding with their 
students?” 

Many factors that prevent even the best teachers from teaching have become clearly visible 
over the last ten years. Some of these impediments to achieving our National Education Goals 
for all of our citizens include these frightening facts from the 1994 National Commission on 
Children report. Imagine in your mind — 

1 Nearly T 3 riiillibn children live in pdverty,' Over'2 million more thari a decade'’ 
ago. 

• At least one in six children has no health care at all. 

• Only slightly more than half of U. S. preschoolers have been fully immunized. 

• On any given night, at least 100,000 children are homeless. 

• Every year, more than a million young people join the ranks of runaways. 

• The overall percentage of students graduating from high school between 1985 
and 1990 decreased for whites, African Americans and Hispanics. 

• Dropouts are 3.5 times more likely to be arrested than high school graduates. 
Secretary Riley (1995) says that 82% of all people behind bars are high school 
drop outs. Drop outs are six times more likely to become unmarried parents. 
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♦ Every year, approximately 1 million teenage girls become pregnant. Births to 
single teens increased 16 % from 1986 to 1991. Data on Figure 1 indicate the 
increase in birth rates for white adolescents from 1940-1992. (Wynne & 
Weinstien, 1995). 

♦ The juvenile violent crime arrest rate increased by 3(X)% from .1960 to 1988. 

♦ 135, (XX) American students bring guns to school everyday. 

♦ Driving after drinking remains the number one killer of adolescents. 

♦ Suicide is now the second leading cause of death among adolescents, and has 
almost tripled since the 1960s. 

♦ Homicide is now the leading cause of death among minority youth age 15 to 
19 and is increasing for white youths. 

♦ Reported child abuse increased 48 % from 1986 to 1991. 

♦ Fifty percent of America’s adults are functionally illiterate. 

We need multiple approaches from many perspectives to solve these immense social- 
educational problems. In many parts of the country, anxious communities are forming 



efiligRtenedTartnerships among health, welfare, juvenile justice, and education systems to pool 
insights and resources. Leadership and funding are coming from school districts, from such 
organizations as the Children’s Aid Society, and from the DeWitt Wallace, Annie Casey, 
Danforth, Hogg, Stuart, and Ford Foundations. Many pilot projects providing school-linked 
comprehensive services are already under way; others are being planned. 

Why We Need School-Linked Comprehensive Services 

The educational enterprise is huge and diverse, and for the majority of our citizens it 
functions very well, but far too many of our children are at risk of failing, dropping out, and 
never realizing their potential or their dreams. 

Six years ago, I was supervising a student teacher in a third-grade classroom of an inner-city 
Houston school. A child came into the classroom with silent tears sliding down his cheeks. 
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With a few questions and a look into his mouth, the teacher realized that the boy had a painfully 
abscessed tooth. The child’s distress captured everyone’s attention. Instruction stopped. 

Many of the children from this poor and crime-ridden neighborhood do not come to school 
ready to learn. Admonitions to teachers to “Put Learning First,’’ can’t be executed under these 
conditions. Unfortunately, for the children in this Houston school, social services and health 
services are a bus trip away. What is more, the parents in this particular case could not be 
located. It is not unusual for teachers in this school to have parent permission slips on hand so 
that they may take children to their own dentist for emergency treatment. In the absence of 
services linked to the school, even though it is not in their job description, dedicated teachers 
become social workers or health care workers. The teachers, principal, and parents work many 
hours to provide the best possible instruction for the children. 

Understand that even though this school was in the center of the drug and crime belt, it was 
better than most other inner-city schools. It had the “I Have a Dream” program and many after- 
school dedicated volunteer tutors. What it lacked was integrated, school-linked, health and social 
services that would allow the teachers to teach. 

People skilled in health and dental care and in child and family counseling are needed to 
attend to the children who come to school in pain or deeply disturbed and not ready to learn. 
While many professionals, agencies, and politicians have strong opinions about the effects on 
learning of the lack of such services, little research in schools has included the crucial variables: 
the health and social welfare of the children and their families. 

Three years ago, I visited two special schools in McAllen, Texas. One served 75 young 
women and their babies. I sat beside a 12-year-old girl, Tiwana, and her 2- week old infant as she 
did her arithmetic. She was getting her problems right. At that school, the young women 
received their basic courses, plus classes in parenting, child development, job skill training, 
and/or college prep. The school provided the young students and their preschool children with 
all of their health and social services at the school site. The principal was a nurse educator. The 
result was that the recidivism rate for second unplanned babies was very low, and the school 
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boasted one of the highest graduation rates in the city. The core of teachers who provided 
courses at this school also taught at an alternative high school a few blocks away, so I went to 
visit it. 

The alternative high school offered a means for drop-outs to reenter the school system and 
graduate. I asked one young woman, who had earlier attempted suicide, how this school was 
different for her. She said, “Adults listen, there aren’t bells jerking you around, and I can go as 
fast as I want to through the work.” After a year when she had wasted her life, she had 
completed two years of course work in one. She caught up with her classmates, would graduate, 
and was planning for college. 

Early this March, I went back to McAllen, curious to see whether the two schools continued 
to thrive. I found that they had been joined into one new Options School serving 200 students. 
Options students represent a balance of racial, social, and economic groups from the McAllen 
high school district. Their common bond: They did not experience success in traditional 
educational settings. All came from high-risk situations that needed to be addressed, such as 
family dysfunction, homelessness, marriage, pregnancy, substance abuse, or combinations of 
these. 

I spoke with Lori, a 17-year-old mother whose daughter is 2 years old. Lori is committed to 
supporting herself and her daughter, even though ]i£r father offers financial help. As a result, her 
schedule resembles that of many graduate students. She goes to school from 8:00 a.m. until 
noon, goes to work as a dental receptionist from 1:00-5:00 p.m., picks up her daughter at 5:30 
p.m., goes to night class from 8:00 to 10:00 p.m., then studies for the next day. She intends to 
enter training as a dental hygienist in the summer. She said, “I was dropping out to support my 
daughter until Options’ flexible schedule and services made school and work possible. My little 
girl and I are going to make it!” And they will! 

Angel, 18 years old, told me of his struggle to support a wife and two children, and attend 
school, with no assistance from either set of parents. At the large high school, he felt as though 
none of the teachers or counselors cared if he stayed in school or not. He tried to attend school 
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and work also, but the school made no accommodation for his schedule. The only jobs he could 
get and go to school too, were at night, from 8 p.m. to 6 a.m. He fell asleep in class. Worse, the 
work paid below minimum wage, not nearly enough to support his family. He applied to 
Options but they were over enrolled. He could not get in. He was on the road to making easy 
money driving hot cars across the border when he was scared straight by an officer of the law. 

He envisioned himself in jail, his wife and children with no support. He went begging to Options 
to accept him, and they did. He is now a leader and speaks eloquently for the program. He will 
graduate in December 1995. And Tiwana, she is 15 and her baby is 3. They both are thriving at 
Options. 

The Options staff provides comprehensive services to meet its students’ varied needs. 
Counselors offer students personal and group counseling and present various activities and 
programs during each day’s advisory period. A social worker is on hand to serve as a liaison for 
the student with family and community. A school nurse assists in detection, treatment and 
referral. Student services such as smoking cessation classes, health screening, and prenatal 
classes are provided for students who need them. Job opportunities and training are provided 
through the Job Training Partnership Act and through cooperative programs administered by the 
school’s career/technology department. Child care is also available while students are attending 
classes. 

These services provided on the school site makes it possible for the Options students to 
work, raise their children, attend school and graduate. Because the services are overlapping, it is 
difficult to use traditional research designs to evaluate the effectiveness of the programs. Dr. 
Knight will describe new approaches to new questions of effectiveness. 
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